Noise reduction using sequential use of multiple noise models

ABSTRACT

Embodiments of the present disclosure relate to performing noise reduction on an input image by first filtering the input image based on coarse noise models of pixels and then subsequently filtering the filtered input image based on finer noise models. The finer noise models use the same or more number of neighboring pixels than the coarse noise filters. The first filtering and subsequent filtering of a pixel in the input image use Mahalanobis distances between the pixel and its neighboring pixels. By performing iterations of filtering using more refined noise models, the noise reduction in the input image can be performed more efficiently and effectively.

BACKGROUND

Image data captured by an image sensor or received from other datasources is often processed in an image processing pipeline beforefurther processing or consumption. For example, raw image data may becorrected, filtered, or otherwise modified before being provided tosubsequent components such as a video encoder. To perform corrections orenhancements for captured image data, various components, unit stages ormodules may be employed.

Such an image processing pipeline may be structured so that correctionsor enhancements to the captured image data can be performed in anexpedient way without consuming other system resources. Although manyimage processing algorithms may be performed by executing softwareprograms on a central processing unit (CPU), execution of such programson the CPU would consume significant bandwidth of the CPU and otherperipheral resources as well as increase power consumption. Hence, imageprocessing pipelines are often implemented as a hardware componentseparate from the CPU and dedicated to perform one or more imageprocessing algorithms.

One of processes of the image processing pipeline is noise reduction.Noise to an image data can be introduced during various operations suchas image capturing, transmission, and transformation. The nature of thenoise removal problem depends on the type of the noise corrupting theimage data, and different types of linear and nonlinear filteringmethods are often used to reduce noise in image data. Linear filters arenot able to effectively eliminate impulse noise as they have a tendencyto blur the edges of an image. On the other hand nonlinear filters aresuited for dealing with impulse noise.

SUMMARY

Embodiments relate to performing noise reduction on image data by usingmultiple noise models. A first noise model is built for a pixel in aninput image using pixel values of first neighboring pixels in the inputimage. First Mahalanobis distances between the pixel and the firstneighboring pixels are computed based on the first noise model.Filtering is performed on the pixel based on the first Mahalanobisdistances to obtain a first filtered image. A second noise model isbuilt for the pixel using pixel values of second neighboring pixels.Second Mahalanobis distances between the pixel and the secondneighboring pixels are computed based on the second noise model.Filtering is performed on the first filtered image based on the secondMahalanobis distances to obtain a second filtered image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level diagram of an electronic device, according to oneembodiment

FIG. 2 is a block diagram illustrating components in the electronicdevice, according to one embodiment.

FIG. 3 is a block diagram illustrating image processing pipelinesimplemented using an image signal processor, according to oneembodiment.

FIG. 4 is a block diagram illustrating a portion of the image processingpipeline including a multiple band noise reduction circuit, according toone embodiment.

FIG. 5 is a conceptual diagram illustrating recursively sub-bandsplitting an input image, according to one embodiment.

FIG. 6 is a block diagram of a multiple band noise reduction circuit,according to one embodiment.

FIG. 7 is a flowchart illustrating a method of performing noisereduction using multiple noise models, according to one embodiment.

The figures depict, and the detail description describes, variousnon-limiting embodiments for purposes of illustration only.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,the described embodiments may be practiced without these specificdetails. In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

Embodiments of the present disclosure relate to performing noisereduction on an input image by first filtering the input image based oncoarse noise models of pixels and then subsequently filtering thefiltered input image based on finer noise models. The finer noise modelsuse the same or more number of neighboring pixels than the coarse noisefilters. The first filtering and subsequent filtering of a pixel in theinput image use Mahalanobis distances between the pixel and itsneighboring pixels. By performing iterations of filtering using morerefined noise models, the noise reduction in the input image can beperformed more efficiently and effectively.

Neighboring pixels of a pixel described herein refers to a set of pixelsthat within a predetermined spatial distance from the pixel. Forexample, neighboring pixels may be 8 pixels adjacent to the pixel (i.e.,spatial distance is 1) or 24 pixels adjacent to the pixel or one pixelspaced apart from the pixel (i.e., spatial distance is 2).

Exemplary Electronic Device

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as personaldigital assistant (PDA) and/or music player functions. Exemplaryembodiments of portable multifunction devices include, withoutlimitation, the iPhone®, iPod Touch®, Apple Watch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. Other portable electronic devices,such as wearables, laptops or tablet computers, are optionally used. Insome embodiments, the device is not a portable communications device,but is a desktop computer or other computing device that is not designedfor portable use. In some embodiments, the disclosed electronic devicemay include a touch sensitive surface (e.g., a touch screen displayand/or a touch pad). An example electronic device described below inconjunction with FIG. 1 (e.g., device 100) may include a touch-sensitivesurface for receiving user input. The electronic device may also includeone or more other physical user-interface devices, such as a physicalkeyboard, a mouse and/or a joystick.

FIG. 1 is a high-level diagram of an electronic device 100, according toone embodiment. Device 100 may include one or more physical buttons,such as a “home” or menu button 104. Menu button 104 is, for example,used to navigate to any application in a set of applications that areexecuted on device 100. In some embodiments, menu button 104 includes afingerprint sensor that identifies a fingerprint on menu button 104. Thefingerprint sensor may be used to determine whether a finger on menubutton 104 has a fingerprint that matches a fingerprint stored forunlocking device 100. Alternatively, in some embodiments, menu button104 is implemented as a soft key in a graphical user interface (GUI)displayed on a touch screen.

In some embodiments, device 100 includes touch screen 150, menu button104, push button 106 for powering the device on/off and locking thedevice, volume adjustment buttons 108, Subscriber Identity Module (SIM)card slot 110, head set jack 112, and docking/charging external port124. Push button 106 may be used to turn the power on/off on the deviceby depressing the button and holding the button in the depressed statefor a predefined time interval; to lock the device by depressing thebutton and releasing the button before the predefined time interval haselapsed; and/or to unlock the device or initiate an unlock process. Inan alternative embodiment, device 100 also accepts verbal input foractivation or deactivation of some functions through microphone 113. Thedevice 100 includes various components including, but not limited to, amemory (which may include one or more computer readable storagemediums), a memory controller, one or more central processing units(CPUs), a peripherals interface, an RF circuitry, an audio circuitry,speaker 111, microphone 113, input/output (I/O) subsystem, and otherinput or control devices. Device 100 may include one or more imagesensors 164, one or more proximity sensors 166, and one or moreaccelerometers 168. The device 100 may include components not shown inFIG. 1.

Device 100 is only one example of an electronic device, and device 100may have more or fewer components than listed above, some of which maybe combined into a components or have a different configuration orarrangement. The various components of device 100 listed above areembodied in hardware, software, firmware or a combination thereof,including one or more signal processing and/or application specificintegrated circuits (ASICs).

FIG. 2 is a block diagram illustrating components in device 100,according to one embodiment. Device 100 may perform various operationsincluding image processing. For this and other purposes, the device 100may include, among other components, image sensor 202, system-on-a chip(SOC) component 204, system memory 230, persistent storage (e.g., flashmemory) 228, motion sensor 234, and display 216. The components asillustrated in FIG. 2 are merely illustrative. For example, device 100may include other components (such as speaker or microphone) that arenot illustrated in FIG. 2. Further, some components (such as orientationsensor 234) may be omitted from device 100.

Image sensor 202 is a component for capturing image data and may beembodied, for example, as a complementary metal-oxide-semiconductor(CMOS) active-pixel sensor, a camera, video camera, or other devices.Image sensor 202 generates raw image data that is sent to SOC component204 for further processing. In some embodiments, the image dataprocessed by SOC component 204 is displayed on display 216, stored insystem memory 230, persistent storage 228 or sent to a remote computingdevice via network connection. The raw image data generated by imagesensor 202 may be in a Bayer color filter array (CFA) pattern(hereinafter also referred to as “Bayer pattern”).

Motion sensor 234 is a component or a set of components for sensingmotion of device 100. Motion sensor 234 may generate sensor signalsindicative of orientation and/or acceleration of device 100. The sensorsignals are sent to SOC component 204 for various operations such asturning on device 100 or rotating images displayed on display 216.

Display 216 is a component for displaying images as generated by SOCcomponent 204. Display 216 may include, for example, liquid crystaldisplay (LCD) device or an organic light emitting diode (OLED) device.Based on data received from SOC component 204, display 116 may displayvarious images, such as menus, selected operating parameters, imagescaptured by image sensor 202 and processed by SOC component 204, and/orother information received from a user interface of device 100 (notshown).

System memory 230 is a component for storing instructions for executionby SOC component 204 and for storing data processed by SOC component204. System memory 230 may be embodied as any type of memory including,for example, dynamic random access memory (DRAM), synchronous DRAM(SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM),static RAM (SRAM) or a combination thereof. In some embodiments, systemmemory 230 may store pixel data or other image data or statistics invarious formats.

Persistent storage 228 is a component for storing data in a non-volatilemanner. Persistent storage 228 retains data even when power is notavailable. Persistent storage 228 may be embodied as read-only memory(ROM), NAND or NOR flash memory or other non-volatile random accessmemory devices.

SOC component 204 is embodied as one or more integrated circuit (IC)chip and performs various data processing processes. SOC component 204may include, among other subcomponents, image signal processor (ISP)206, a central processor unit (CPU) 208, a network interface 210, sensorinterface 212, display controller 214, graphics processor (GPU) 220,memory controller 222, video encoder 224, storage controller 226, andvarious other input/output (I/O) interfaces 218, and bus 232 connectingthese subcomponents. SOC component 204 may include more or fewersubcomponents than those shown in FIG. 2.

ISP 206 is hardware that performs various stages of an image processingpipeline. In some embodiments, ISP 206 may receive raw image data fromimage sensor 202, and process the raw image data into a form that isusable by other subcomponents of SOC component 204 or components ofdevice 100. ISP 206 may perform various image-manipulation operationssuch as image translation operations, horizontal and vertical scaling,color space conversion and/or image stabilization transformations, asdescribed below in detail with reference to FIG. 3.

CPU 208 may be embodied using any suitable instruction set architecture,and may be configured to execute instructions defined in thatinstruction set architecture. CPU 208 may be general-purpose or embeddedprocessors using any of a variety of instruction set architectures(ISAs), such as the x86, PowerPC, SPARC, RISC, ARM or MIPS ISAs, or anyother suitable ISA. Although a single CPU is illustrated in FIG. 2, SOCcomponent 204 may include multiple CPUs. In multiprocessor systems, eachof the CPUs may commonly, but not necessarily, implement the same ISA.

Graphics processing unit (GPU) 220 is graphics processing circuitry forperforming graphical data. For example, GPU 220 may render objects to bedisplayed into a frame buffer (e.g., one that includes pixel data for anentire frame). GPU 220 may include one or more graphics processors thatmay execute graphics software to perform a part or all of the graphicsoperation, or hardware acceleration of certain graphics operations.

I/O interfaces 218 are hardware, software, firmware or combinationsthereof for interfacing with various input/output components in device100. I/O components may include devices such as keypads, buttons, audiodevices, and sensors such as a global positioning system. I/O interfaces218 process data for sending data to such I/O components or process datareceived from such I/O components.

Network interface 210 is a subcomponent that enables data to beexchanged between devices 100 and other devices via one or more networks(e.g., carrier or agent devices). For example, video or other image datamay be received from other devices via network interface 210 and bestored in system memory 230 for subsequent processing (e.g., via aback-end interface to image signal processor 206, such as discussedbelow in FIG. 3) and display. The networks may include, but are notlimited to, Local Area Networks (LANs) (e.g., an Ethernet or corporatenetwork) and Wide Area Networks (WANs). The image data received vianetwork interface 210 may undergo image processing processes by ISP 206.

Sensor interface 212 is circuitry for interfacing with motion sensor234. Sensor interface 212 receives sensor information from motion sensor234 and processes the sensor information to determine the orientation ormovement of the device 100.

Display controller 214 is circuitry for sending image data to bedisplayed on display 216. Display controller 214 receives the image datafrom ISP 206, CPU 208, graphic processor 220 or system memory 230 andprocesses the image data into a format suitable for display on display216.

Memory controller 222 is circuitry for communicating with system memory230. Memory controller 222 may read data from system memory 230 forprocessing by ISP 206, CPU 208, GPU 220 or other subcomponents of SOCcomponent 204. Memory controller 222 may also write data to systemmemory 230 received from various subcomponents of SOC component 204.

Video encoder 224 is hardware, software, firmware or a combinationthereof for encoding video data into a format suitable for storing inpersistent storage 128 or for passing the data to network interface w10for transmission over a network to another device.

In some embodiments, one or more subcomponents of SOC component 204 orsome functionality of these subcomponents may be performed by softwarecomponents executed on ISP 206, CPU 208 or GPU 220. Such softwarecomponents may be stored in system memory 230, persistent storage 228 oranother device communicating with device 100 via network interface 210.

Image data or video data may flow through various data paths within SOCcomponent 204. In one example, raw image data may be generated from theimage sensor 202 and processed by ISP 206, and then sent to systemmemory 230 via bus 232 and memory controller 222. After the image datais stored in system memory 230, it may be accessed by video encoder 224for encoding or by display 116 for displaying via bus 232.

In another example, image data is received from sources other than theimage sensor 202. For example, video data may be streamed, downloaded,or otherwise communicated to the SOC component 204 via wired or wirelessnetwork. The image data may be received via network interface 210 andwritten to system memory 230 via memory controller 222. The image datamay then be obtained by ISP 206 from system memory 230 and processedthrough one or more image processing pipeline stages, as described belowin detail with reference to FIG. 3. The image data may then be returnedto system memory 230 or be sent to video encoder 224, display controller214 (for display on display 216), or storage controller 226 for storageat persistent storage 228.

Example Image Signal Processing Pipelines

FIG. 3 is a block diagram illustrating image processing pipelinesimplemented using ISP 206, according to one embodiment. In theembodiment of FIG. 3, ISP 206 is coupled to image sensor 202 to receiveraw image data. ISP 206 implements an image processing pipeline whichmay include a set of stages that process image information fromcreation, capture or receipt to output. ISP 206 may include, among othercomponents, sensor interface 302, central control 320, front-endpipeline stages 330, back-end pipeline stages 340, image statisticsmodule 304, vision module 322, back-end interface 342, and outputinterface 316. ISP 206 may include other components not illustrated inFIG. 3 or may omit one or more components illustrated in FIG. 3.

In one or more embodiments, different components of ISP 206 processimage data at different rates. In the embodiment of FIG. 3, front-endpipeline stages 330 (e.g., raw processing stage 306 and resampleprocessing stage 308) may process image data at an initial rate. Thus,the various different techniques, adjustments, modifications, or otherprocessing operations performed by these front-end pipeline stages 330at the initial rate. For example, if the front-end pipeline stages 330process 2 pixels per clock cycle, then raw processing stage 306operations (e.g., black level compensation, highlight recovery anddefective pixel correction) may process 2 pixels of image data at atime. In contrast, one or more back-end pipeline stages 340 may processimage data at a different rate less than the initial data rate. Forexample, in the embodiment of FIG. 3, back-end pipeline stages 340(e.g., noise processing stage 310, color processing stage 312, andoutput rescale 314) may be processed at a reduced rate (e.g., 1 pixelper clock cycle). Although embodiments described herein includeembodiments in which the one or more back-end pipeline stages 340process image data at a different rate than an initial data rate, insome embodiments back-end pipeline stages 340 may process image data atthe initial data rate.

Sensor interface 302 receives raw image data from image sensor 202 andprocesses the raw image data into an image data processable by otherstages in the pipeline. Sensor interface 302 may perform variouspreprocessing operations, such as image cropping, binning or scaling toreduce image data size. In some embodiments, pixels are sent from theimage sensor 202 to sensor interface 302 in raster order (i.e.,horizontally, line by line). The subsequent processes in the pipelinemay also be performed in raster order and the result may also be outputin raster order. Although only a single image sensor and a single sensorinterface 302 are illustrated in FIG. 3, when more than one image sensoris provided in device 100, a corresponding number of sensor interfacesmay be provided in ISP 206 to process raw image data from each imagesensor.

Front-end pipeline stages 330 process image data in raw or full-colordomains. Front-end pipeline stages 330 may include, but are not limitedto, raw processing stage 306 and resample processing stage 308. A rawimage data may be in Bayer raw format, for example. In Bayer raw imageformat, pixel data with values specific to a particular color (insteadof all colors) is provided in each pixel. In an image capturing sensor,image data is typically provided in a Bayer pattern. Raw processingstage 306 may process image data in a Bayer raw format.

The operations performed by raw processing stage 306 include, but arenot limited, sensor linearization, black level compensation, fixedpattern noise reduction, defective pixel correction, raw noisefiltering, lens shading correction, white balance gain, and highlightrecovery. Sensor linearization refers to mapping non-linear image datato linear space for other processing. Black level compensation refers toproviding digital gain, offset and clip independently for each colorcomponent (e.g., Gr, R, B, Gb) of the image data. Fixed pattern noisereduction refers to removing offset fixed pattern noise and gain fixedpattern noise by subtracting a dark frame from an input image andmultiplying different gains to pixels. Defective pixel correction refersto detecting defective pixels, and then replacing defective pixelvalues. Raw noise filtering refers to reducing noise of image data byaveraging neighbor pixels that are similar in brightness. Highlightrecovery refers to estimating pixel values for those pixels that areclipped (or nearly clipped) from other channels. Lens shading correctionrefers to applying a gain per pixel to compensate for a dropoff inintensity roughly proportional to a distance from a lens optical center.White balance gain refers to providing digital gains for white balance,offset and clip independently for all color components (e.g., Gr, R, B,Gb in Bayer format). Components of ISP 206 may convert raw image datainto image data in full-color domain, and thus, raw processing stage 306may process image data in the full-color domain in addition to orinstead of raw image data.

Resample processing stage 308 performs various operations to convert,resample, or scale image data received from raw processing stage 306.Operations performed by resample processing stage 308 may include, butnot limited to, demosaic operation, per-pixel color correctionoperation, Gamma mapping operation, color space conversion anddownscaling or sub-band splitting. Demosaic operation refers toconverting or interpolating missing color samples from raw image data(for example, in a Bayer pattern) to output image data into a full-colordomain. Demosaic operation may include low pass directional filtering onthe interpolated samples to obtain full-color pixels. Per-pixel colorcorrection operation refers to a process of performing color correctionon a per-pixel basis using information about relative noise standarddeviations of each color channel to correct color without amplifyingnoise in the image data. Gamma mapping refers to converting image datafrom input image data values to output data values to perform specialimage effects, including black and white conversion, sepia toneconversion, negative conversion, or solarize conversion. For the purposeof Gamma mapping, lookup tables (or other structures that index pixelvalues to another value) for different color components or channels ofeach pixel (e.g., a separate lookup table for Y, Cb, and Cr colorcomponents) may be used. Color space conversion refers to convertingcolor space of an input image data into a different format. In oneembodiment, resample processing stage 308 converts RBD format into YCbCrformat for further processing.

Central control module 320 may control and coordinate overall operationof other components in ISP 206. Central control module 320 performsoperations including, but not limited to, monitoring various operatingparameters (e.g., logging clock cycles, memory latency, quality ofservice, and state information), updating or managing control parametersfor other components of ISP 206, and interfacing with sensor interface302 to control the starting and stopping of other components of ISP 206.For example, central control module 320 may update programmableparameters for other components in ISP 206 while the other componentsare in an idle state. After updating the programmable parameters,central control module 320 may place these components of ISP 206 into arun state to perform one or more operations or tasks. Central controlmodule 320 may also instruct other components of ISP 206 to store imagedata (e.g., by writing to system memory 230 in FIG. 2) before, during,or after resample processing stage 308. In this way full-resolutionimage data in raw or full-color domain format may be stored in additionto or instead of processing the image data output from resampleprocessing stage 308 through backend pipeline stages 340.

Image statistics module 304 performs various operations to collectstatistic information associated with the image data. The operations forcollecting statistics information may include, but not limited to,sensor linearization, mask patterned defective pixels, sub-sample rawimage data, detect and replace non-patterned defective pixels, blacklevel compensation, lens shading correction, and inverse black levelcompensation. After performing one or more of such operations,statistics information such as 3A statistics (Auto white balance (AWB),auto exposure (AE), auto focus (AF)), histograms (e.g., 2D color orcomponent) and any other image data information may be collected ortracked. In some embodiments, certain pixels' values, or areas of pixelvalues may be excluded from collections of certain statistics data(e.g., AF statistics) when preceding operations identify clipped pixels.Although only a single statistics module 304 is illustrated in FIG. 3,multiple image statistics modules may be included in ISP 206. In suchembodiments, each statistic module may be programmed by central controlmodule 320 to collect different information for the same or differentimage data.

Vision module 322 performs various operations to facilitate computervision operations at CPU 208 such as facial detection in image data. Thevision module 322 may perform various operations includingpre-processing, global tone-mapping and Gamma correction, vision noisefiltering, resizing, keypoint detection, convolution and generation ofhistogram-of-orientation gradients (HOG). The pre-processing may includesubsampling or binning operation and computation of luminance if theinput image data is not in YCrCb format. Global mapping and Gammacorrection can be performed on the pre-processed data on luminanceimage. Vision noise filtering is performed to remove pixel defects andreduce noise present in the image data, and thereby, improve the qualityand performance of subsequent computer vision algorithms. Such visionnoise filtering may include detecting and fixing dots or defectivepixels, and performing bilateral filtering to reduce noise by averagingneighbor pixels of similar brightness. Various vision algorithms useimages of different sizes and scales. Resizing of an image is performed,for example, by binning or linear interpolation operation. Keypoints arelocations within an image that are surrounded by image patches wellsuited to matching in other images of the same scene or object. Suchkeypoints are useful in image alignment, computing cameral pose andobject tracking. Keypoint detection refers to the process of identifyingsuch keypoints in an image. Convolution may be used in image/videoprocessing and machine vision. Convolution may be performed, forexample, to generate edge maps of images or smoothen images. HOGprovides descriptions of image patches for tasks in mage analysis andcomputer vision. HOG can be generated, for example, by (i) computinghorizontal and vertical gradients using a simple difference filter, (ii)computing gradient orientations and magnitudes from the horizontal andvertical gradients, and (iii) binning the gradient orientations.

Back-end interface 342 receives image data from other image sources thanimage sensor 102 and forwards it to other components of ISP 206 forprocessing. For example, image data may be received over a networkconnection and be stored in system memory 230. Back-end interface 342retrieves the image data stored in system memory 230 and provide it toback-end pipeline stages 340 for processing. One of many operations thatare performed by back-end interface 342 is converting the retrievedimage data to a format that can be utilized by back-end processingstages 340. For instance, back-end interface 342 may convert RGB, YCbCr4:2:0, or YCbCr 4:2:2 formatted image data into YCbCr 4:4:4 colorformat.

Back-end pipeline stages 340 processes image data according to aparticular full-color format (e.g., YCbCr 4:4:4 or RGB). In someembodiments, components of the back-end pipeline stages 340 may convertimage data to a particular full-color format before further processing.Back-end pipeline stages 340 may include, among other stages, noiseprocessing stage 310 and color processing stage 312. Back-end pipelinestages 340 may include other stages not illustrated in FIG. 3.

Noise processing stage 310 performs various operations to reduce noisein the image data. The operations performed by noise processing stage310 include, but are not limited to, color space conversion,gamma/de-gamma mapping, temporal filtering, noise filtering, lumasharpening, and chroma noise reduction. The color space conversion mayconvert an image data from one color space format to another color spaceformat (e.g., RGB format converted to YCbCr format). Gamma/de-gammaoperation converts image data from input image data values to outputdata values to perform special image effects. Temporal filtering filtersnoise using a previously filtered image frame to reduce noise. Forexample, pixel values of a prior image frame are combined with pixelvalues of a current image frame. Noise filtering may include, forexample, spatial noise filtering. Luma sharpening may sharpen lumavalues of pixel data while chroma suppression may attenuate chroma togray (i.e. no color). In some embodiment, the luma sharpening and chromasuppression may be performed simultaneously with spatial nose filtering.The aggressiveness of noise filtering may be determined differently fordifferent regions of an image. Spatial noise filtering may be includedas part of a temporal loop implementing temporal filtering. For example,a previous image frame may be processed by a temporal filter and aspatial noise filter before being stored as a reference frame for a nextimage frame to be processed. In other embodiments, spatial noisefiltering may not be included as part of the temporal loop for temporalfiltering (e.g., the spatial noise filter may be applied to an imageframe after it is stored as a reference image frame (and thus is not aspatially filtered reference frame).

Color processing stage 312 may perform various operations associatedwith adjusting color information in the image data. The operationsperformed in color processing stage 312 include, but are not limited to,local tone mapping, gain/offset/clip, color correction,three-dimensional color lookup, gamma conversion, and color spaceconversion. Local tone mapping refers to spatially varying local tonecurves in order to provide more control when rendering an image. Forinstance, a two-dimensional grid of tone curves (which may be programmedby the central control module 320) may be bi-linearly interpolated suchthat smoothly varying tone curves are created across an image. In someembodiments, local tone mapping may also apply spatially varying andintensity varying color correction matrices, which may, for example, beused to make skies bluer while turning down blue in the shadows in animage. Digital gain/offset/clip may be provided for each color channelor component of image data. Color correction may apply a colorcorrection transform matrix to image data. 3D color lookup may utilize athree dimensional array of color component output values (e.g., R, G, B)to perform advanced tone mapping, color space conversions, and othercolor transforms. Gamma conversion may be performed, for example, bymapping input image data values to output data values in order toperform gamma correction, tone mapping, or histogram matching. Colorspace conversion may be implemented to convert image data from one colorspace to another (e.g., RGB to YCbCr). Other processing techniques mayalso be performed as part of color processing stage 312 to perform otherspecial image effects, including black and white conversion, sepia toneconversion, negative conversion, or solarize conversion.

Output rescale module 314 may resample, transform and correct distortionon the fly as the ISP 206 processes image data. Output rescale module314 may compute a fractional input coordinate for each pixel and usesthis fractional coordinate to interpolate an output pixel via apolyphase resampling filter. A fractional input coordinate may beproduced from a variety of possible transforms of an output coordinate,such as resizing or cropping an image (e.g., via a simple horizontal andvertical scaling transform), rotating and shearing an image (e.g., vianon-separable matrix transforms), perspective warping (e.g., via anadditional depth transform) and per-pixel perspective divides applied inpiecewise in strips to account for changes in image sensor during imagedata capture (e.g., due to a rolling shutter), and geometric distortioncorrection (e.g., via computing a radial distance from the opticalcenter in order to index an interpolated radial gain table, and applyinga radial perturbance to a coordinate to account for a radial lensdistortion).

Output rescale module 314 may apply transforms to image data as it isprocessed at output rescale module 314. Output rescale module 314 mayinclude horizontal and vertical scaling components. The vertical portionof the design may implement series of image data line buffers to holdthe “support” needed by the vertical filter. As ISP 206 may be astreaming device, it may be that only the lines of image data in afinite-length sliding window of lines are available for the filter touse. Once a line has been discarded to make room for a new incomingline, the line may be unavailable. Output rescale module 314 maystatistically monitor computed input Y coordinates over previous linesand use it to compute an optimal set of lines to hold in the verticalsupport window. For each subsequent line, output rescale module mayautomatically generate a guess as to the center of the vertical supportwindow. In some embodiments, output rescale module 314 may implement atable of piecewise perspective transforms encoded as digital differenceanalyzer (DDA) steppers to perform a per-pixel perspectivetransformation between a input image data and output image data in orderto correct artifacts and motion caused by sensor motion during thecapture of the image frame. Output rescale may provide image data viaoutput interface 316 to various other components of device 100, asdiscussed above with regard to FIGS. 1 and 2.

In various embodiments, the functionally of components 302 through 342may be performed in a different order than the order implied by theorder of these functional units in the image processing pipelineillustrated in FIG. 3, or may be performed by different functionalcomponents than those illustrated in FIG. 3. Moreover, the variouscomponents as described in FIG. 3 may be embodied in variouscombinations of hardware, firmware or software.

Example Pipelines Associated with Multiple Band Noise Reduction Circuit

FIG. 4 is a block diagram illustrating a portion of the image processingpipeline including a multiple band noise reduction (MBNR) circuit 420,according to one embodiment. In the embodiment of FIG. 4, MBNR circuit420 is part of a resample processing stage 308 that also includes, amongother components, a scaler 410 and a sub-band splitter circuit 430. Theresample processing stage 308 performs scaling, noise reduction, andsub-band splitting in a recursive manner.

As a result of recursive processing, the resample processing stage 380outputs a series of high frequency component image data HF(N) and lowfrequency component image data LF(N) derived from an original inputimage 420 where N represents the levels of downsampling performed on theoriginal input image 402. For example, HF(0) and LF(0) represent a highfrequency component image data and a low frequency component image datasplit from the original input image 402, respectively, while HF(1) andLF(1) represent a high frequency component image data and a lowfrequency component image data split from a first downscaled version ofthe input image 402, respectively.

MBNR circuit 420 is a circuit that performs noise reduction on multiplebands of the input image 402 by processing progressively downscaledversions of the input image 402. The input image 402 is first passed onthrough a multiplexer 414 to MBNR circuit 420 for noise reduction. Thenoise reduced version 422 of the original input image 402 is generatedby MBNR circuit 420 and fed to a sub-band splitter 430. The sub-bandsplitter 430 splits the noise reduced 422 version of the original inputimage 402 into the high frequency component image data HF(0) and the lowfrequency component image data LF(0). The high frequency component imagedata HF(0) is passed onto a sub-band processing pipeline 448 and then toa sub-band merger 352. In contrast, the low frequency component imageLF(0) is passed through a demultiplexer 440 and is fed back to theresample processing stage 308 for downscaling by a scaler 410.

The scaler 410 generates a downscaled version 412 of the low frequencycomponent image LF(0) fed to the scaler 410, and passes it onto MBNRcircuit 420 via the multiplexer 414 for noise reduction. MBNR circuit420 performs noise reduction to generate a noise reduced version 432 ofthe downscaled image 412 and sends it to the sub-band splitter 430 toagain split the processed low frequency image data LF(0) into the highfrequency component image data HF(1) and the low frequency componentimage data LF(1). The high frequency component image data HF(1) is sentto the sub-band processing pipeline 448 and then the sub-band merger 352whereas the low frequency component image data LF(1) is again fed backto the scaler 410 to repeat the process within the resample processingstage 308. The process of generating a high frequency component imagedata HF(N) and a low frequency component image data LF(N) is repeateduntil the final level of band-splitting is performed by the sub-bandsplitter 430. When the final level of band-splitting is reached, the lowfrequency component image data LF(N) is passed through the demultiplexer440 and a multiplexer 446 to the sub-band processing pipeline 448 andthe sub-band merger 352.

FIG. 5 is a conceptual diagram illustrating recursively sub-bandsplitting the original input image 402, according to one embodiment. Inthe example of FIG. 5, the input image 402 is sub-band split 6 times bythe resample processing stage 308. First, the input image 402 at thebottom of FIG. 5 splits into HF(1) and LF(1), which undergoes noisereduction process and again splits into HF(2) and LF(2), which againundergoes noise reduction process and splits into HF(3) and LF(3), andso on. The sub-band components HF(1) through HF(6) and LF(6) are passedon from the resample processing stage 308 to the sub-band processingpipeline 448.

As described above, MBNR circuit 420 performs noise reduction on theinput image 402 as well as its downscaled low frequency versions of theinput image 402. This enables MBNR circuit 420 to perform noisereduction on multiple bands of the original input image 402. It is to benoted, however, that only a single pass of noise reduction may beperformed on the input image 402 by MBNR circuit 420 without sub-bandsplitting and scaling.

Referring back to FIG. 4 in the context of FIG. 5, HF(1) through HF(6)and LF(6) are processed by the sub-band processing pipeline 448 andpassed onto the sub-band merger 352. The sub-band merger 352 mergesprocessed high frequency component image data HF(N)′ and processed lowfrequency component image data LF(N)′ to generate a processed LF(N-1)′.The processed LF(N-1)′ is then fed back to the sub-band merger 352 viathe demultiplexer 450 and the multiplexer 446 for merging with theprocessed HF (N-1)′ to generate a processed LF(N-2)′. The process ofcombining the processed high frequency component image data and theprocessed low frequency component data is repeated until the sub-bandmerger 352 generates a processed version 454 of input image that isoutputted via the demultiplexer 450.

Example Architecture of Multiple Band Noise Reduction Circuit

FIG. 6 is a block diagram illustrating MBNR circuit 420, according toone embodiment. MBNR circuit 420 receive an input image data 602 andgenerates noise reduced image data 670 and Mahalanobis distances 660.MBNR circuit 420 may include, among other components, a first reversalprocessing circuit 606, a coarse noise model circuit 616, a firstphotometric kernel calculator circuit 626, a first bilateral filtercircuit 636, a second reversal processing circuit 642, a fine noisemodel circuit 646, a second photometric kernel calculator circuit 656and a second bilateral filter circuit 668. MBNR circuit 420 usesMahalanobis distances of pixel values instead of Euclidean distances ofpixel values to perform bilateral filtering, as described below indetail. The use of Mahalanobis distances is advantageous, among otherreasons, because it yields better noise reduction performance than usingEuclidean distances.

The first reversal processing circuit 606 is a circuit that reversesconversions and processing previously performed on raw image data. Thereversed processes may also include, for example, prior linear andnon-linear color space transformations (e.g., converting YCC format backto a raw image in Bayer pattern format) and lens shading correction. Asa result, the reversal processing circuit 616 generates a raw image data610, which may be in RGB color space. The reversal of image processingprocesses enables the subsequent processes to be performed without anydistortions or noises introduced by previous image processing processes.If the input image data is a raw image, then the processing by thereversal processing circuit 606 may be omitted.

The coarse noise model circuit 616 generates a coarse noise model foreach pixel in the raw image data 610. In one embodiment, the coarsenoise model for a pixel is a covariance matrix at the correspondingpixel location, which is a function of a vector of a true pixel values(e.g., red, green, blue values) of the pixel location. The true pixelvalues are unknown, and hence, the pixel values at the pixel location inthe raw image data 610 are assumed as the true pixel values for thecovariance matrix. There may be no cross covariance between the pixelvalues of the raw image data, and therefore, the covariance matrix canbe diagonal with the main diagonal including individual R, G, and Bvariances. The R, G and B variances for the pixel location can becalculated from a first number of pixels adjacent to the pixel location.In one embodiment, the first number of pixels are 8 pixels adjacent tothe pixel location or 24 pixel within two pixel distances from the pixellocation. After generating a coarse noise model for a pixel location,the coarse noise model circuit 606 proceeds to generate a coarse noisemodel for the next pixel location. The coarse noise models 620 are thensent to the first Mahalanobis photometric kernel calculator circuit 626.

The first photometric kernel calculator circuit 626 computes Mahalanobisdistances for the pixels based on the coarse noise models 620 generatedby the coarse noise model circuit 616. Mahalanobis distance MD betweentwo vectors can be computed as:MD=√{square root over (Δ^(T)Σ⁻¹Δ)}  Equation 1where Δ represents a difference between vectors and Σ represents acovariance matrix of the noise. In one embodiment, calculation ofMahalanobis distance can be simplified by transforming two vectors to acolor space where covariance matrix is diagonal. The transformation canbe done by the first reversal processing circuit 606.

The first bilateral filter circuit 636 performs the bilateral filteringon the input image data 602 using the photometric coefficients 630received from the first photometric kernel calculator circuit 626. Inone embodiment, the bilateral filtering performs the computationaccording to the following equation, which combines spatial andphotometric kernels into one adaptive kernel:

                                      Equation  2${\overset{\rightarrow}{y}\left\lbrack {i,j} \right\rbrack} = \frac{n = {{\frac{\sum\limits_{{- N} + 1}^{\frac{N - 1}{2}}}{2}m} = {\frac{\sum\limits_{M + 1}^{\frac{M - 1}{2}}}{2}{W_{p}\left\lbrack {n + m} \right\rbrack} \times {W_{S}\left\lbrack {n,m} \right\rbrack} \times {\overset{\rightarrow}{x}\left\lbrack {{i - n},{j - m}} \right\rbrack}}}}{n = {{\frac{\sum\limits_{{- N} + 1}^{\frac{N - 1}{2}}}{2}m} = {\frac{\sum\limits_{M + 1}^{\frac{M - 1}{2}}}{2}{W_{p}\left\lbrack {n,m} \right\rbrack} \times {W_{S}\left\lbrack {n,m} \right\rbrack}}}}$where {right arrow over (y)} represents filtered pixels values, Nrepresents a horizontal support size of the bilateral filter, Mrepresents a horizontal support size of the bilateral filter, W_(P)represents coefficients of photometric kernels that are functions of thefirst Mahalanobis distances, W_(S) represents coefficients of spatialkernels that can be different for different pixel components luma andchroma, {right arrow over (x)} represents pixel values of the image data610 (which can be in (YUV, YCbCr or RGB format), and i and j are currentpixel indexes.

In one embodiment, photometric distances can be used to calculatecoefficients 630 of a photometric filter kernel (hereinafter alsoreferred to as “photometric coefficients”). The equation for thephotometric coefficients 630 are as follows:W _(p) [n,m]=G(MD[n,m])  Equation 3where MD[n, m] represents a Mahalanobis distance between the currentpixel and a [n, m] pixel in its vicinity; G represents any non-linearfunction (usually Gaussian). In one embodiment, the photometriccoefficients can be computed as follows:W _(p) [n,m]=1−min(1,Tmp×Slope)  Equation 4Tmp=max(0,k[n,m]MD[n,m]−Knee)  Equation 5where Knee and Slope are function parameters, and k[n, m] represents aspatial adjustment coefficient.

The second reversal processing circuit 642 performs the same functionand operations as the first reversal processing circuit 606 except thatthe second reversal processing circuit 606 provides the reverted data644 to the fine noise model 646 instead of the coarse noise model 616.

Instead of using the filtered pixel values of the first bilateral filter636 as the final result of MBNR circuit 420, the filtered pixel values640 are fed to a fine noise model circuit 646 and subsequent circuitsfor another iteration of more refined filtering. Specifically, the finenoise model circuit 646 is fed with the reverted data 644 and thefiltered pixel values 640.

Based on the reverted data 644 and the filtered pixel values 640, thefine noise model circuit 646 generates the fine noise models. For apixel location, the fine noise models can be generated using a secondnumber of neighboring pixels. The second number of neighboring pixel canbe the same number of neighboring circuits as the first number ofneighboring circuits (used in the coarse noise model circuit 616) or canbe more than the first number of neighboring circuits. In order togenerate the fine noise models, the fine noise model circuit 646 assumesthat the filtered pixel values 640 are the true pixel values.

The second photometric kernel calculator circuit 656 computesMahalanobis distances 660 and the photometric kernels 662 for thefiltered pixel values 640 based on the fine noise models 650 generatedby the fine noise model circuit 646. Other than the use of the reverteddata 644 instead of the raw input pixel data and the use of the finenoise models 650 instead of the coarse noise models 620, the secondphotometric kernel circuit 656 performs in the same way as the firstphotometric kernel calculator circuit 626 to generate photometriccoefficients 662, and therefore, the detailed description thereof isomitted herein for the sake of brevity.

The second bilateral filter circuit 668 performs the bilateral filteringon the input image data 602 using the photometric coefficients 662received from the second photometric kernel calculator 656. Other thanthe use of the photometric kernels generated from the second photometriccoefficients 660, the operation and the function of the second bilateralfilter circuit 668 are the same as the first bilateral filter circuit636, and therefore, the detailed description thereof is omitted hereinfor the sake of brevity. The second bilateral filter circuit 668 outputsthe noise reduced version 422 of the raw image data 610 as the output ofthe MBNR circuit 420.

In one embodiment, the Mahalanobis distances 660 can also be output fromMBNR circuit 420. The Mahalanobis distances 660 may be used, forexample, by the sub-band splitter circuit 430 to identify relationshipsbetween pixels in the noise reduced version 422 of the raw image data610.

Although the embodiment of FIG. 6 shows the filtering the raw image datain two stages, three or more stages of bilateral filtering based ongradually finer noise models may be performed to generate the noisereduced version of the raw image data.

Example Process for Performing Noise Reduction

FIG. 7 is a flowchart illustrating a method of performing noisereduction using multiple noise models, according to one embodiment. Thereversal processing circuit 606 reverses 710 at least part of imageprocessing performed on raw pixel data received from an image sensor202. The revised image processing may include, among other processes,prior linear and non-linear color space transformations and lens shadingcorrection.

The coarse noise model circuit 616 builds 720 coarse noise models forpixels in an input image using pixel values of first neighboring pixelsin the input image. The first Mahalanobis distance computation circuit626 determines 730 the first Mahalanobis distances between pixels andtheir first neighboring pixels based on the first noise models.

The first bilateral filter 636 performs 740 filtering on the pixelsbased on the first Mahalanobis distances to obtain filtered pixelvalues. The fine noise model circuit 646 builds 750 fine noise modelsfor the pixels using pixel values of second neighboring pixels. Thesecond Mahalanobis distance computation circuit 656 determines 760 thesecond Mahalanobis distances between the pixels and the secondneighboring pixels based on the second noise models.

The second bilateral filter circuit 668 performs 770 filtering on thefirst filtered pixel values based on the second Mahalanobis distances toobtain the noise reduced version 422 of the image data.

The process as described above with reference to FIG. 7 is merelyillustrative. For example, the process of reversing 710 the imageprocessing may be omitted if the input image data is in a raw imageformat. Moreover, additional processing may be performed on the imagedata, such as performing another stage of building finer noise models,determining Mahalanobis distances based on the finer noise models andperforming filtering based on such Mahalanobis distances.

What is claimed is:
 1. An apparatus for processing image data,comprising: a first noise model circuit configured to build a firstnoise model for a pixel in an input image using pixel values of firstpixels neighboring the pixel in the input image; a first distancecomputation circuit coupled to the first noise model circuit andconfigured to compute first Mahalanobis distances between the pixel andthe first neighboring pixels based on the first noise model; a firstfilter coupled to the first distance computation circuit and configuredto perform filtering on the pixel based on the first Mahalanobisdistances to obtain a first filtered image; a second noise model circuitcoupled to the first filter and configured to build a second noise modelfor the pixel based on second pixels neighboring the pixel in the firstfiltered image; a second distance computation circuit coupled to thesecond noise model circuit and configured to compute second Mahalanobisdistances between the pixel and the second neighboring pixels based onthe second noise model; and a second filter coupled to the seconddistance computation circuit and configured to perform filtering on theinput image based on the second Mahalanobis distances to obtain a secondfiltered image.
 2. The apparatus of claim 1, wherein a number of thesecond neighboring pixels are more than or equal to a number of thefirst neighboring pixels.
 3. The apparatus of claim 1, furthercomprising a first reversal processing circuit configured to reverse atleast part of image processing performed on raw pixel data from an imagecapturing device to generate the input image.
 4. The apparatus of claim3, further comprising a second reversal processing circuit configured toreverse at least part of image processing performed on the raw pixeldata to obtain a reverted image provided to the second noise modelcircuit for building the second noise model.
 5. The apparatus of claim3, wherein the reversed image processing comprises color spacetransformation and lens shading correction.
 6. The apparatus of claim 1,wherein the first filter is a bilateral filter that uses a first numberof pixel values, and the second filter is a bilateral filter that uses asecond number of pixel values more than the first number of pixelvalues.
 7. The apparatus of claim 1, further comprising: a sub-bandsplitter circuit coupled to the second filter to receive the secondfiltered image, the sub-band splitter configured to split the secondfiltered image into a first frequency data and a second frequency dataof a lower frequency than the first frequency data.
 8. The apparatus ofclaim 7, further comprising: a demultiplexer having an input coupled toan output of the sub-band splitter circuit to receive the secondfrequency data, the demultiplexer having a first output coupled to asubsequent processing circuit and a second output; and a scaler circuitcoupled to the second output of the demultiplexer to receive the secondfrequency data, the scaler circuit configured to generate a downscaledversion of the second frequency data as a third frequency data sent tothe first noise model circuit for processing.
 9. The apparatus of claim7, wherein the second distance computation circuit is configured to sendthe second Mahalanobis distances to the sub-band splitter circuit, andwherein the sub-band splitter circuit is configured to identifyrelationships between pixels in the second filtered image based at leaston the second Mahalanobis distances.
 10. The apparatus of claim 1,wherein the first distance computation circuit is configured to computephotometric coefficients based on the first Mahalanobis distances forperforming filtering at the first filter, and the second distancecomputation circuit is configured to compute photometric coefficientsbased on the second Mahalanobis distances for performing filtering atthe second filter.
 11. The apparatus of claim 1, wherein the input imagehas subpixels arranged in a Bayer pattern.
 12. A method of processingimage data, comprising: building a first noise model for a pixel in aninput image using pixel values of first pixels neighboring the pixel inthe input image; computing first Mahalanobis distances between the pixeland the first neighboring pixels based on the first noise model;performing filtering on the pixel based on the first Mahalanobisdistances to obtain a first filtered image; building a second noisemodel for the pixel based on second pixels neighboring the pixel in thefirst filtered image; computing second Mahalanobis distances between thepixel and the second neighboring pixels based on the second noise model;and performing filtering on the input image based on the secondMahalanobis distances to obtain a second filtered image.
 13. The methodof claim 12, wherein a number of the second neighboring pixels are morethan or equal to a number of the first neighboring pixels.
 14. Themethod of claim 12, further comprising reversing at least part of imageprocessing performed on raw pixel data from an image capturing device togenerate the input image.
 15. The method of claim 14, wherein thereversed image processing comprises color space transformation and lensshading correction.
 16. The method of claim 15, further comprisingreversing at least part of image processing performed on the raw pixeldata for building the second noise model.
 17. The method of claim 12,wherein performing filtering on the pixel based on the first Mahalanobisdistances comprises performing bilateral filtering using a first numberof pixel values, and wherein performing filtering on the first filteredpixel value comprises performing bilateral filtering using a secondnumber of pixel values more than the first number of pixel values. 18.The method of claim 12, wherein computing the first Mahalanobisdistances comprises computing photometric coefficients based on thefirst Mahalanobis distances for performing filtering at the firstfilter, and wherein computing photometric coefficients based on thesecond Mahalanobis distances for performing filtering at the secondfilter.
 19. The method of claim 12, wherein the input image hassubpixels arranged in a Bayer pattern.
 20. A resampling circuit in animage processing pipeline, comprising: a first noise model circuitconfigured to build a first noise model for a pixel in an input imageusing pixel values of first pixels neighboring the pixel in the inputimage; a first distance computation circuit coupled to the first noisemodel circuit and configured to compute first Mahalanobis distancesbetween the pixel and the first neighboring pixels based on the firstnoise model; a first filter coupled to the first distance computationcircuit and configured to perform filtering on the pixel based on thefirst Mahalanobis distances to obtain a first filtered image; a secondnoise model circuit coupled to the first filter and configured to builda second noise model for the pixel based on second pixels neighboringthe pixel in the first filtered image; a second distance computationcircuit coupled to the second noise model circuit and configured tocompute second Mahalanobis distances between the pixel and the secondneighboring pixels based on the second noise model; and a second filtercoupled to the second distance computation circuit and configured toperform filtering on the input image based on the second Mahalanobisdistances to obtain a second filtered image.